1. Field of the Invention
This invention relates to methods, programs, and systems for generating test cases for performing testing of computer systems or the like.
2. Description of Related Art
Computer systems and control apparatuses have various settable parameters, and behaviors thereof change depending on combinations of those parameters. For example, as parameters used in a computer system, there are the type of an operating system to be introduced, the type of a web browser to be used, the type of a network card, the capacities of a main memory and a hard disk, the type of a central processing unit (CPU), the number of cores thereof, and so forth.
Testing of such systems and apparatuses is desirably designed so that all combinations of parameters are covered. However, since the number of test cases exponentially increases against the number of parameters when all combinations of parameters are tested, such testing is not allowed in terms of processing time. Accordingly, methods for generating a set of test cases for performing efficient testing using fewer test cases have been hitherto proposed.
As one example, pair-wise testing, i.e., testing using pairs, is known. In pair-wise testing, a set of test cases is generated so that all possible pairs of values can be covered for given two parameters. Pair-wise testing is based on knowledge that many bugs are triggered by combinations of a few parameters.
For example, when there are n parameters serving as test items, nC2 cases are present for selecting two parameters from the n parameters in pair-wise testing. When there are m1 and m2 possible values for the two selected parameters k1 and k2, respectively, the number of pairs of values is equal to nC2×m1×m2.
Meanwhile, the following are known as patent literatures of the related art of this field.
Japanese Unexamined Patent Application Publication No. 2003-256206 discloses a method for assisting creation of a test plan. The method includes a step of setting quality information of each program module of a software system, a step of estimating first man-hours spent for detecting bugs of the program module from the quality information, a step of creating module-specification correspondence information representing a correspondence between the module and specifications extracted from a specification document of the software system, a step of estimating second man-hours for detecting bugs of the specifications from the first man-hours and the module-specification correspondence information, and a step of creating a test plan for distributing available test resources to tests of the software system based on the second man-hours.
Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2007-535723 discloses a testing tool including an “automated multidimensional traceability matrix” system for determining linkages between interrelated system components, means for identifying a change in one or more of the interrelated system components, means for applying the “automated multidimensional traceability matrix”, means for executing all of or a subset of test scenarios associated with the interrelated system components that may be affected by the change, and means for evaluating the results of the executed test scenarios.
Japanese Patent Application No. 2003-406478 discloses a method for monitoring a bus with pair-wise participants. The method includes detecting a problem during a transaction between first and second participants on the bus, and determining which participant is at fault for this problem or whether this problem includes a systemic bus problem.
Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2010-532894 discloses a system and a method for assigning pieces of content to time-slot samples for measuring effects of the assigned content. By means of the system and the method, pair-wise content relatedness data is received that compares each of the pieces of content with the other pieces of content and identifies the piece of content as an experimental content piece or control content piece, and the experimental or control content pieces are algorithmically assigned to time-slot samples using the content relatedness data, and additional content pieces assigned to a particular time-slot sample exclude non-identical related experimental content pieces defined relative to experimental content pieces previously assigned to the particular time-slot sample.
As described above, in the related art, various practical-level testing methods including the pair-wise method are known.
Meanwhile, the inventors have encountered a situation where a set of test cases already exists before starting testing, such as the aforementioned one. The set of existing test cases includes test cases, validity of which has been already verified manually or in other ways. Accordingly, the inventors have considered that it might be advantageous if a reasonably sized set of test cases including this result can be acquired using the set of existing test cases.
However, the above-described testing methods of the related art do not suggest a technique for generating test cases using the set of existing test cases when the set of test cases is acquired.